Motivated by recent observations that suggest a low density of old stars around the Milky Way supermassive black hole, models for the nuclear star cluster are considered that have not yet reached a steady state under the influence of gravitational encounters. A core of initial radius 1-1.5 pc evolves to a size of approximately 0.5 pc after 10 Gyr, roughly the size of the observed core. The absence of a Bahcall-Wolf cusp is naturally explained in these models, without the need for fine-tuning or implausible initial conditions. In the absence of a cusp, the time for a 10-solar-mass black hole to spiral in to the Galactic center from an initial distance of 5 pc can be much greater than 10 Gyr. Assuming that the stellar black holes had the same phase-space distribution initially as the stars, their density after 5-10 Gyr is predicted to rise very steeply going into the stellar core, but could remain substantially below the densities inferred from steady-state models that include a steep density cusp in the stars. Possible mechanisms for the creation of the parsec-scale initial core include destruction of stars on centrophilic orbits in a pre-existing triaxial nucleus, inhibited star formation near the supermassive black hole, or ejection of stars by a massive binary. The implications of these models are discussed for the rates of gravitational-wave inspiral events, as well as other physical processes that depend on a high density of stars or stellar mass black holes near Sagittarius A*.